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EXECUTIVE  SUMMARY 


Problem: 

A  new  Air  Force  instruction,  AFI 63-124,  Performance-Based  Service  Contracts,  effective  since 
1  Apr  99,  offers  a  unique  surveillance  approach  for  Air  Force  service  contracts.  The  surveillance 
approach,  using  contractor  metrics,  bears  the  philosophy  to  provide  “insight”  rather  than 
“oversight”  in  order  to  enhance  the  quality  assurance  role.  The  philosophy  also  centers  on 
continuous  improvement  within  performance  management.  Air  Force  contracting  agencies  have 
little  experience  with  this  new  focus,  and  particularly  with  using  contractor  metrics  in  contract 
quality  assurance.  At  the  same  time,  functional  areas  are  losing  their  organic  expertise  to 
conduct  appropriate  surveillance.  There  is  also  no  current  method  to  consistently  evaluate  the 
health  of  a  contracting  program. 

As  a  result,  HQ  PACAF/LG  asked  AFLMA  to  develop  standardized  contractor  metrics  for 
service  contracts.  SAF/AQC  cosponsored  out  of  concern  of  evaluating  Air  Force  service 
contracting  efforts.  In  response  to  these  requests,  our  objectives  identify  top-level  metrics  to 
determine  the  Air  Force  service  contracts  program’s  health  and  provide  the  necessary  Service 
Delivery  Summaries  and  Quality  Assurance  Surveillance  Plans  related  to  using  contractor 
metrics. 

Objective(s): 

This  project  identifies  contractor  metrics  for  Air  Force  service  contracts.  The  study  focuses  on 
five  major  service  contracts:  “The  Big  4”  (custodial,  grounds  maintenance,  Military  Family 
Housing  (MFH)  maintenance,  and  refuse  collection  and  recycling)  plus  full  food  service.  It  then 
includes  general  metrics  for  all  service  contracts.  The  project  addresses  the  following  objectives: 

•  Providing  metrics  definitions,  the  data,  and  the  objective  each  metric  accomplishes; 

•  Outlining  rules  about  getting  the  data; 

•  Identifying  the  appropriate  data  source(s); 

•  Creating  Service  Delivery  Summary  and  Quality  Assurance  Surveillance  Plan  for  each 
service  contract. 

Analysis/Results: 

In  order  to  strategically  choose  and  craft  the  most  appropriate  metrics,  this  study  first  suggests  a 
vision  and  strategy  for  Air  Force  service  contracts.  The  vision: 

Long-term,  best  value  service  providers,  as  active  Air  Force  partners,  focus  on  the 
customer  and  continuous  improvement  for  maximum  achievement,  and  address 
Air  Force  needs  while  also  complying  with  applicable  Air  Force  instructions  and 
policy  directives  while  also  employing  best  commercial  practices. 
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Key  objectives  were  then  established  to  steer  these  service  performance  areas  towards  the  vision. 
These  objectives  include  identifying  customer  complaint  causes,  establishing  a  healthy 
workforce  vitality,  promoting  safety,  and  advocating  contractor  suggestions. 

The  study  advocates  Air  Force-chosen  metrics  rather  than  contractor-suggested  metrics  for 
measuring  the  service  contract’s  and  partnership’s  health.  Note:  the  study’s  scope  does  not 
address  internal,  quality  control  metrics  which  a  contractor  would  develop.  With  respect  to 
several  objectives,  they  focus  on  specific,  measurable,  meaningful,  and  practical  indicators  for 
highly  effective  partnerships.  The  study  also  addresses  using  customer  complaints.  For 
sufficient  feedback,  information  must  show  reasons  for  customer  dissatisfaction,  not  necessarily 
the  number  of  customer  complaints.  A  high  number  of  non- valid  customer  complaints  were 
recurring  problems.  As  non- valid  complaints  could  represent  a  problem  area,  QAs  have 
indicated  their  preference  to  quantify  the  percentage  of  non- valid  complaints  of  the  total 
complaints  received.  Data  integrity  becomes  suspect  when  the  customer  complaint  system 
includes  more  people. 

Note:  The  metrics  developed  as  a  result  of  this  study  are  listed  in  the  appendices  (Appendix  A.l 
through  Appendix  G.2).  The  metrics  are  presented  in  a  useful  format  to  contracting  and  quality 
assurance  personnel.  The  same  appendices  are  provided  in  electronic  form  on  the  World  Wide 
Web  at  the  Business  Solutions  Exchange  web  site.  Here  is  the  address:  www.bsx.org,  and  then 
go  to  <BSX  Communities  of  Interest>  and  then  <Contractor  Metrics  for  Service  Contracts>. 

This  web  site  contains  the  following: 

•  The  Service  Delivery  Summary  (SDS)  for  each  service  contract  in  this  study. 

•  The  Quality  Assurance  Surveillance  Plan  that  correspond  to  each  SDS. 

•  Excel  workbooks  to  illustrate  how  to  collect  the  information  and  to  portray  the  metrics. 

•  An  outline  of  necessary  steps  towards  using  contractor  metrics  as  a  surveillance  method. 

•  A  “Frequently  Asked  Questions”  document  has  been  posted  there  as  well  to  cover  many 
issues  regarding  this  surveillance  approach. 

•  The  Office  of  Federal  Procurement  Policy’s  (OFPP’s)  memorandum  which  provides  a 
performance-based  service  contracts  checklist. 

Conclusions: 

The  metrics  created  in  this  study  (located  in  the  Appendices)  are  portions  of  actual  Service 
Delivery  Summaries  (SDSs)  and  Quality  Assurance  Surveillance  Plans  (QASPs)  for  the  Air 
Force’s  five  major  service  contracts.  Functional  commanders,  contracting  commander,  and 
contracting  officers  may  use  these  metrics  for  their  service  contracts.  They  may  also  tailor  them 
to  suit  local  needs. 
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CHAPTER  1 


INTRODUCTION 


BACKGROUND 


Senior  federal  officials  have  emphasized  performance-based  contracting  since  1991,  but  recently 
they  have  increased  their  attention  on  the  approach.1  This  is  partially  due  to  the  recent  attention 
on  using  best  commercial  practices,  consolidating  requirements,  operating  within  constrained 
resources,  and  reducing  total  ownership  costs.2  For  example,  the  Under  Secretary  of  Defense 
cited  the  necessity  to  shift  from  supply  management  to  supplier  management.3, 4  Also,  the  1999 
Contracting  21  Business  Plan  highlights  contract  performance.5  The  business  plan  advocates 
developing  and  managing  effective  service  relationships  in  order  to  fulfill  high  expectations.6  As 
another  example,  the  HQ  AMC/CC  even  announced  full  support  to  achieve  better  contractor 
performance  and  reducing  overall  costs  through  performance-based  contracting.7  Also,  a  recent 
interview  with  the  Air  Force  Civil  Engineer  (HQ  USAF/ILE)  also  addressed  the  current  need  to 
fully  support  the  mission  with  high  quality  of  life  levels  by  maintaining  and  operating  Air  Force 
installations.8 

Senior  officials  have  also  emphasized  the  business  advisor  role  for  contracting  officers.  The 
advisors  need  objective,  performance-based,  and  customer-focused  information  to  facilitate 
performance  management.  The  business  advisor  uses  this  information  as  a  tool  to  implement  the 
necessary  changes  for  achieving  maximum  performance. 

Within  the  Air  Force,  Air  Force  Instruction  63-124,  Performance-Based  Service  Contracts 
(implemented  1  April  1999)  features  new  methods  for  the  business  advisor  and  his  or  her  team.9 
The  instruction  also  involves  new  methods  for  Quality  Assurance  (QA)  personnel.10  The  new 
methods  shift  from  oversight  (in-process  inspections  by  random  sampling)  to  insight  (validating 
a  provider’s  management  system  and  process  performance  metrics)  for  quality  and  contract 
compliance.  To  facilitate  insight,  performance-based  specifications  include  measurable 
performance  objectives  to  encourage  contractors  to  develop  innovative  solutions  and  implement 
cost-effective  methods.  The  QA  validates  the  performance  objectives’  achievement  and 
promotes  continuous  improvement.  With  reliance  on  customer  complaints,  the  QA  may  use 
contractor  metrics  in  addition  to  other  surveillance  methods  to  measure  the  contractor’s 
performance  quality.11 

The  Federal  Acquisition  Regulation  (FAR)  requires  contractors  to  maintain  an  inspection  system 
for  their  services.12, 13  As  for  acceptance,  the  Air  Force  may  review  the  contractor’s  inspections 
to  confirm  contract  compliance.14, 15 

These  inspections  may  provide  information  to  diverse  stakeholders.  The  AFI  directs  various 
management  levels  to  provide  updates  to  their  higher  management  level  about  their  services 
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contracts.16  That  is,  MAJCOM  directors  shall  update  their  MAJCOM/CCs  annually  on  the 
command’s  service  contract  program.  A  Performance  Management  Council’s  (PMC’s)  duties 
include  assessing  their  contractors’  operation  effectiveness  and  Air  Force  contract  management 
effectiveness  in  addition  to  approving  partnership  agendas.  Contracting  squadron  commanders 
shall  update  their  installation  commanders  twice  each  year  on  the  installation’s  contract  services 
program.  A  Business  Requirements  Advisory  Group’s  (BRAG’s)  duties  include  contract 
performance  management  (e.g.,  analyzing  contractor  metrics  and  evaluating  performance  for 
payment  and  award  fees). 

Throughout  these  various  reporting  levels,  performance  management  centers  around  identifying 
problems,  providing  solutions,  and  offering  approaches  towards  continuous  improvement. 
Continuous  improvement  (using  feedback  to  make  ongoing  adjustments  in  pursuing  a  corporate 
vision)  includes  information  to  determine  whether  or  not  objectives  have  been  achieved.17  The 
partnership  communicates  to  correct  or  reward  those  activities  and  processes  which  affect 
achievement  towards  the  vision. 

Performance  management  systems  exist  to  accomplish  several  other  efforts.  They  may  include: 
a)  Demonstrating  performance;  b)  Providing  information  to  different  stakeholders  who  have 
different  expectations;  c)  Determining  if  they  have  fulfilled  certain  needs;  d)  Achieving 
maximum  performance  within  constrained  budgets  and  resources;  e)  Identifying  performance 
level  and  comparing  it  to  others;  f)  Determining  effective  resource  use;  g)  Setting  goals;  h) 
Learning  through  root  cause  analysis;  and  i)  Standardizing  improvements.18, 19'20 

Air  Force/contractor  partnerships  require  these  efforts.  To  their  aid,  the  AFI  63-124, 
Performance-Based  Service  Contracts  allows  a  new  surveillance  method:  reviewing  contractor 
metrics.  This  method  shifts  inspection  responsibility  to  the  service  contractors  and  adjusts 
Quality  Assurance  inspection  duties  with  validating  those  service  contractors’  quality  control 
plans  and  promoting  continuous  improvement.  Validation  duties  include  enough  surveillance  to 
know  if  the  metrics  portray  accurate  information.  Although  new  to  the  Air  Force  (AFI  63-124 
became  effective  1  Apr  99),  commercial  practices  have  been  using  Total  Quality  Management 
initiatives  for  quite  some  time.  Still,  industry  and  government  organizations  incorporate 
performance  management  to  demonstrate  return  on  investment,  benchmark,  and  facilitate 
continuous  improvement. 


PROBLEM  STATEMENT 


A  new  Air  Force  instruction,  AFI  63-124,  Performance-Based  Service  Contracts,  effective  since 
1  Apr  99,  offers  a  unique  surveillance  approach  for  Air  Force  service  contracts.  The  surveillance 
approach,  using  contractor  metrics,  bears  the  philosophy  to  provide  “insight”  rather  than 
“oversight”  in  order  to  enhance  the  quality  assurance  role.  The  philosophy  also  centers  on 
continuous  improvement  within  performance  management.  Air  Force  contracting  agencies  have 
little  experience  with  this  new  focus,  and  particularly  with  using  contractor  metrics  in  contract 
quality  assurance.  At  the  same  time,  functional  areas  are  losing  their  organic  expertise  to 
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conduct  appropriate  surveillance.  There  is  also  no  current  method  to  consistently  evaluate  the 
health  of  a  contracting  program. 

As  a  result,  HQ  PACAF/LG  asked  AFLMA  to  develop  standardized  contractor  metrics  for 
service  contracts.  SAF/AQC  cosponsored  out  of  concern  of  evaluating  Air  Force  service 
contracting  efforts.  In  response  to  these  requests,  our  objectives  identify  top-level  metrics  to 
determine  the  Air  Force  service  contracts  program’s  health  and  provide  the  necessary  Service 
Delivery  Summaries  and  Quality  Assurance  Surveillance  Plans  related  to  using  contractor 
metrics. 


STUDY  OBJECTIVES 


This  project  identifies  contractor  metrics  for  Air  Force  service  contracts.  The  study  focuses  on 
five  major  service  contracts:  “The  Big  4”  (custodial,  grounds  maintenance,  Military  Family 
Housing  (MFH)  maintenance,  and  refuse  collection  and  recycling)  plus  full  food  service  (and 
mess  attendants).  It  then  includes  general  metrics  for  all  service  contracts.  The  project  addresses 
the  following  objectives: 

•  Providing  metrics  definitions,  the  data,  and  the  objective  each  metric  accomplishes; 

•  Outlining  rules  about  getting  the  data; 

•  Identifying  the  appropriate  data  source(s); 

•  Creating  Service  Delivery  Summary  and  Quality  Assurance  Surveillance  Plan  for  each 
service  contract. 
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CHAPTER  2 


METHODOLOGY 


APPROACH 


Interviews  were  conducted  with  commercial  and  government  representatives  who  have  a  broad 
range  of  facility  management  responsibilities.  These  interviews  explored  a  wide  range  of 
performance  management  methods.  Specific  attention  was  drawn  to  scorecard  systems  and 
quality  control  plans  to  learn  how  they  best  use  performance  management.  After  developing 
contractor  metrics  for  Air  Force  service  contracts,  a  close  review  of  metric  collection  and 
reporting  details  followed. 

This  study  addresses  five  service  contracts: 

•  Custodial 

•  Full  food  service  (and  mess  attendants) 

•  Grounds  maintenance 

•  Military  Family  Housing  (MFH)  maintenance 

•  Refuse  and  recycling  collection 

These  contracts  represent  five  most  common  Air  Force  service  requirements:  four  civil 
engineering  contracts  (“The  Big  Four”)  and  the  full  food  service  contracts.  The  latter  was  added 
due  to  recent  attention  in  implementing  A  FI  63-124,  Performance-Based  Service  Contracts. 
Commercial  and  public  service  facilities  managers  also  commonly  outsource  housekeeping,  food 
service,  grounds  keeping,  trash  collection,  and  building  maintenance,  among  other  services.21 
The  study  also  provides  general  metrics  for  all  Air  Force  service  contracts,  regardless  of  the 
service  requirement  (See  Appendix  A.l  and  Appendix  A.2). 

For  this  study’s  metrics,  see  the  Appendices.  They  are  in  the  Service  Delivery  Summary  and 
Quality  Assurance  Surveillance  Plan  format,  which  is  useful  to  the  contracting  officers  and 
quality  assurance  personnel. 

VISION  AND  STRATEGY 

Before  raising  attention  to  developing  metrics,  this  study  first  suggests  a  vision  and  strategy  for 
Air  Force  service  contracts.  Otherwise,  one  may  create  metrics  without  knowing  how  they  affect 
a  strategic  plan.  After  all,  performance  management  aligns  vision,  strategic  objectives,  and 
performance  outcomes  and  strategic  objectives  integrate  a  business  plan’s  outcomes.22, 23  The 
vision  and  strategy  for  Air  Force  service  contracts: 
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Long-term,  best  value  service  providers,  as  active  Air  Force  partners,  focus  on  the 
customer  and  continuous  improvement  for  maximum  achievement,  and  address 
Air  Force  needs  while  also  complying  with  applicable  Air  Force  instructions  and 
policy  directives  while  also  employing  best  commercial  practices. 

This  vision  was  suggested  with  regard  to  how  a  partnership  conducts  performance  management. 
It  starts  with  selecting  a  best  value  contractor,  which  is  the  most  preferred  method  of  selecting 
the  most  advantageous  contractor  proposal  for  the  service  requirement.  (“Best  value”  techniques 
employ  a  tradeoff  between  proposal  merits,  the  contractor’s  past  performance,  cost/price,  and 
proposal  risk.  Some  “best  value”  methods,  such  as  Performance  Price  Tradeoffs  only  consider 
past  performance  and  cost/price,  but  still  the  end  result  is  choosing  the  most  advantageous 
contractor  on  other  factors  besides  cost/price  alone.) 

Assuming  that  the  best  contractor  has  been  selected,  the  contract  relationship  should  continue 
through  a  typical  5-year  service  contract  length.  The  vision  underscores  the  need  for  a 
partnership  in  order  to  address  Air  Force  business  plans  and  goals.  The  current  overarching 
philosophy  is  to  deem  the  contract  relationship  as  a  partnership.  This  vision  uses  the  word 
“active”  to  reinforce  the  point  rather  than  considering  partnerships  as  “business  as  usual.”  AFI 
63-124,  Performance-Based  Service  Contracts  emphasizes  customer  satisfaction  and  customer 
feedback  as  well  as  relying  heavily  on  commercial  practices  while  still  complying  with  Air  Force 
instructions  and  policy  directives. 

Performance  areas  within  each  service  contract  were  then  identified  to  answer  “What  contract 
requirements  achieve  this  vision?”  Key  objectives  were  then  established  to  steer  these  service 
performance  areas  towards  the  vision.  For  example,  these  objectives  include  identifying 
customer  complaint  causes,  establishing  a  healthy  workforce  vitality,  promoting  safety,  and 
advocating  contractor  suggestions. 


CRITERIA  FOR  METRICS 


Several  criteria  for  the  metrics  were  used.  One  was  using  effective  measurements.  In  other 
words,  indicators  should  measure  the  achievement  of  goals.  Functional  representatives 
suggested  or  agreed  to  the  indicators.  Equally  important,  the  metrics  should  focus  on  outcomes 
and  results,  not  on  the  process.  After  all,  the  contractor  bears  responsibility  for  process  metrics. 
The  metrics  proposed  in  this  study  facilitate  the  business  advisor’s  strategic  plan  for  fulfilling  the 
service  requirements.  Overall,  these  metrics  demonstrate  whether  or  not  one  has  achieved 
desired  results,  they  may  identify  trends,  they  drive  appropriate  action,  and  they  lead  one  to  fact- 
based  decisions.  Another  criterion  was  efficiency.  Metrics  were  chosen  with  respect  to  the  time 
and  effort  required  to  collect,  analyze,  and  evaluate  them  (i.e.,  cost  versus  benefit).  Finally,  the 
metrics  were  filtered  through  the  following  additional  criteria:  risk  involved  in  not  having  the 
metric,  objective  priority,  contract  attention/visibility,  and  any  pre-existing  requirements  to 
record  the  data  on  an  Air  Force  form  (the  last  criterion  avoids  creating  additional  work). 
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The  metrics  were  also  limited  in  number  to  facilitate  a  better  focus  and  to  emphasize  their 
importance.  However,  too  few  measures  would  have  the  potential  to  portray  an  inaccurate 
performance  assessment.  In  this  regard,  insufficient  information  may  cause  erroneous 
conclusions.  Therefore,  the  report  proposes  limited,  yet  sufficient,  metrics. 

Some  criteria  found  in  Malcolm-Baldridge  evaluations  for  performance  excellence  for  services 
were  also  used.24  These  considerations  included  incorporating  new  technology,  process  design, 
cycle  time,  key  performance  measures,  and  suggestions.  Partnership  criteria  also  reflect 
actionable  feedback,  targets,  minimizing  total  costs,  and  a  supportive  customer  role.  Therefore, 
some  metrics  reflect  reasons  for  customer  dissatisfaction,  contractor  responsiveness  to  the 
customer,  and  customer  training. 


SAMPLE 


This  study  involved  more  than  110  people  through  face-to-face  interviews,  e-mails,  and/or  phone 
calls.  These  people  represented  9  public-sector  organizations  (other  than  PACAF  organizations) 
and  39  commercial  corporations.  Both  the  public  organizations  and  the  commercial  corporations 
were  useful  in  gathering  market  research.  The  interview  method  was  used  with  a  standard 
question  list,  yet  the  interviews  allowed  the  conversation  to  stray  from  the  list  as  well.  Site 
visits,  a  performance  review,  and  a  dining  facility’s  menu  planning  board  were  also  used  to 
solicit  and  define  appropriate  metrics  for  the  five  service  contracts.  Then  in-depth  interviews 
with  the  respective  civil  engineering  and  services  personnel  were  used  to  validate  and  further 
refine  the  metrics.  Face-to-face  interviews  were  held  with  people  representing  the  following  Air 


Force  organizations: 

15  CES 

•  80  CONS 

15  CONS 

•  AFCESA  (and  TRW) 

15  SVS 

•  AFMIA/MICO 

16  CONS 

•  HQ  AETC/LGC 

16  CES 

•  HQ  AMC/LGC 

16  SVS 

•  HQ  PACAF/LGC 

21  CONS 

•  HQ  AFSPC/LGC 

42  ABW/LGC 

•  HQ  USAFA/LGC 

42  ABW/CES 

•  SAF/AQCO 

REVIEW  PERIOD 


Preliminary  metrics  were  e-mailed,  as  a  minimum,  to  the  PACAF  QAPCs  with  courtesy  copies 
to  their  squadron  commanders.  Other  organizations  were  represented  in  the  review,  including: 
SAF/AQCO,  HQ  AMC/LGC,  HQ  ACC/LGC,  HQ  AFSPC/LGC,  HQ  AETC/LGC,  16  CONS,  42 
ABW/CE,  HQ  USAFA/LGC,  RAND,  UMS  Group,  and  AFCESA  (TRW).  The  preliminary 
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metrics  were  also  posted  on  Business  Solutions  Exchange  (BSX)  with  discussion  forums  for  each 
service  contract.  The  review  period  lasted  almost  one  month.  The  following  questions  were 
posed  to  all  people  involved  in  the  review: 

•  What  needs  to  be  addressed  in  standardizing  these  metrics? 

•  What  challenges  are  included  in  standardizing  these  metrics? 

•  Do  you  think  these  metrics  paint  a  full  picture  on  the  health  of  that  contract? 

•  What  objections  do  your  functionals  (and  joint  service  customers)  raise  to  using  these 
metrics? 

•  How  do  you  think  the  local  customs  will  affect  your  service  contractor  using  the 
suggested  metrics? 

The  answers  to  these  questions  were  used  to  rework  and  finalize  the  metrics.  As  stated  in  other 
sections  of  this  report,  the  metrics  from  this  study  are  provided  in  the  Appendices  (Appendix  A.l 
through  Appendix  G.2).  They  are  in  the  Service  Delivery  Summary  and  Quality  Assurance 
Surveillance  Plan  format  and  they  are  also  located  on  in  electronic  format  on  the  World  Wide 
Web.  The  web  site  is:  www.bsx.org,  then  choose  <BSX  Communities  of  Interest>  and  then 
<Contractor  Metrics  for  Service  Contracts>. 


AN  OUTLINE  FOR  DEVELOPING  CONTRACTOR  METRICS 


This  study  provides  metrics  for  five  specific  contracts.  Because  the  metrics  are  specific  to  those 
contracts,  the  following  outline  can  serve  as  a  general  guideline  for  developing  metric  sets  as 
surveillance  methods  for  other  types  of  contracts: 

•  Develop  a  vision  for  the  partnership  and  its  performance  management. 

•  Choose  strategic  objectives  that  steer  specific  performance  areas  towards  this  vision. 

•  Conduct  appropriate  market  research  and  brainstorm  as  necessary  to  identify  potential 
metrics. 

•  Select  metrics  that  indicate  how  well  the  objectives  are  achieved.  Note:  using  carefully 
selected  contractor  metrics  along  with  Air  Force  (customer)  developed  metrics  is  a 
surveillance  methods  that  can  be  accompanied  with  other  methods  such  as  customer 
complaint,  periodic  inspection,  third  party  audit,  and  so  on. 

•  Refine  the  metrics  by  soliciting  the  contractor’s  feedback. 

•  Mutually  agree  on  the  selected  metrics.  Note:  Writing  them  into  the  Quality  Assurance 
Surveillance  Plan  is  a  contracting  officer’s  unilateral  decision  yet  building  the 
contractor’s  buy-in  will  promote  a  more  active  partnership. 

•  Collect  information  for  the  metrics  during  a  baseline  period  (e.g.,  the  first  three  months). 
Expect  to  see  a  learning  curve.  Use  the  optimal  portions  of  the  learning  curve  to  create  a 
baseline  and  then  identify  a  performance  target. 

•  Report  the  information  to  stakeholders:  senior  management,  subordinates,  and 
customers. 
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•  Evaluate  the  selected  metrics  as  necessary  and  either  continue  using,  further  refine,  or 
replace  them  as  necessary.  Be  careful  not  to  modify/replace  metrics  just  because  they  are 
showing  unintended  results.  Also,  use  customer  feedback  in  validating  and/or  evaluating 
metrics.  For  example,  check  to  see  if  the  contractor’s  metrics  are  congruent  with 
customer  feedback  or  customer  complaints. 
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CHAPTER  3 


THE  DATA  AND  RESULTS 

INTRODUCTION 

Note:  The  metrics  developed  as  a  result  of  this  study  are  listed  in  the  appendices  (Appendix  A.l 
through  Appendix  G.2).  The  metrics  are  presented  in  a  useful  format  to  contracting  and  quality 
assurance  personnel  (Service  Delivery  Summary  and  Quality  Assurance  Surveillance  Plan).  The 
same  appendices  are  provided  in  electronic  form  on  the  World  Wide  Web  at  the  Business 
Solutions  Exchange  web  site.  Here  is  the  address:  www.bsx.org,  and  then  go  to  <BSX 
Communities  of  Interest>  and  then  <Contractor  Metrics  for  Service  Contracts>.  In  addition  to 
the  SDSs  and  QASPs,  an  Excel  workbook  has  been  provided  to  illustrate  how  to  collect  the 
information  and  portray  the  metrics  for  each  service  contract  in  this  study.  A  “Frequently  Asked 
Questions”  document  has  been  posted  there  as  well  to  cover  many  issues  regarding  this 
surveillance  approach.  The  OFPP’s  memorandum  which  provides  a  performance-based  service 
contracts  checklist  has  been  provided  as  well. 

Respondents  raised  several  issues.  One  concern  identified  the  difference  between  quality  control 
metrics  and  those  that  the  Air  Force  reviews.  Related  to  this  issue,  they  also  introduced  an 
argument  between  Air  Force-chosen  metrics  and  those  suggested  by  the  contractor.  Respondents 
also  offered  various  opinions  on  whether  or  not  to  link  the  partnership  metrics  to  an  award  fee. 
Other  issues  included  conducting  performance  reviews,  participating  in  partnerships,  explaining 
the  business  advisor  role,  and  using  customer  complaints. 


LIMITATIONS 


Given  time  and  personnel  factors,  this  study  was  limited  in  targeting  only  five  major  service 
contracts.  As  other  service  contracts  would  benefit  from  further  metrics  research,  this  study 
provides  a  process  for  doing  so. 

The  choice  of  implementation  has  its  tradeoffs  of  advantages  and  disadvantages.  This  study’s 
recommendations  do  not  require  modifying  a  contract  or  waiting  until  the  next  award.  Some 
have  perceived  that  incorporating  new  metrics  into  their  contract  would  add  costs  and/or  change 
the  requirements.  However,  Quality  Assurance  Surveillance  Plans  (QASPs)  are  not  a  contractual 
instrument;  they  may  be  changed  at  the  Contracting  Officer’s  (CO’s)  discretion.  Further, 
partnership  agreements  should  allow  for  flexibility  in  performance  management.  For  those  not 
having  an  agreement.  Business  Requirement  and  Advisory  Group  members  (with  the  CO’s  and 
Functional  Director’s  endorsement)  may  introduce  these  metrics  to  their  providers  and  both  sides 
of  the  contract  relationship  may  agree  to  use  them.  If  this  is  not  possible,  at  least  waiting  until 
awarding  a  new  contract  introduces  the  approach  at  the  beginning  of  the  contract  relationship. 
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RESULTS 


Market  research  among  commercial  firms  and  government  organizations  revealed  diverse  sets  of 
metrics.25  The  diversity  represents  the  link  a  provider  must  make  to  the  customer’s  strategic 
goals  and  not  vice-versa.  Because  each  customer/provider  relationship  differs,  standard  metrics 
were  not  found  except  for  a  common  tendency  to  use  employee  turnover  (chum  rate)  and  safety 
indicators.  As  an  intrinsic  standard,  partnerships  normally  contain  a  mix  of  metrics  representing 
results,  outcomes,  processes,  and  impact  statements.26 

One  debate  regards  who  establishes  the  metrics — the  Air  Force  or  the  contractor?  One  side 
argues  the  contractor  may  tailor  the  metric  set  to  the  business  process.  The  contractor-suggested 
metrics  argument  advocates  the  contractor’s  ingenuity  and  industry  knowledge.  With  metrics 
already  tested  and  proven  to  have  sustained  the  contractor  in  business,  this  argument  underscores 
how  a  contractor  would  avoid  additional  work  (and  cost)  to  meet  Air  Force-chosen  metrics. 
Furthermore,  this  argument  recognizes  the  reduced  Air  Force’s  quality  assurance  manpower  and 
therefore  places  the  burden  on  the  contractor  to  develop  metrics  based  on  their  knowledge  in  the 
industry. 

Note:  contractors  do  keep  internal  metrics — ones  related  to  their  resource  allocation,  processes, 
and  policies.  However,  these  metrics  may  differ  from  those  chosen  by  a  customer.  For  example, 
customers  tend  to  measure  whether  or  not  they  received  services  while  providers  track  whether 
their  processes  achieved  appropriate  results  (i.e.,  costs).27  This  study  does  not  standardize 
internal  (quality  control)  metrics;  it  provides  the  information  tools  most  useful  to  the  Air  Force  in 
order  to  facilitate  a  working  partnership,  continuous  improvement,  and  contractor  innovation. 

To  illustrate  the  difference  between  internal  contractor  metrics  and  those  useful  to  the 
partnership,  consider  the  full  food  service  contract.  The  contractor  maintains  many  metrics 
which  the  Air  Force  does  not  necessarily  review.  A  few  examples  include  the  budgeted  versus 
actual  hours  used  for  the  day,  average  number  of  breaks  per  person,  and  the  number  of  breakfast, 
lunch,  and  dinner  meals  by  date  for  forecasting  future  resource  decisions.  The  partnership  does 
not  necessarily  consider  these  metrics.  Instead,  they  consider  overall  operation,  revenue,  and 
sanitation  objectives,  to  name  a  few.  Therefore,  the  partnership  should  consider  certain  metrics 
such  as  a  monthly  Hennessy  program  score,  monthly  earned  income  (i.e.,  sales),  and  percent  of 
managers  who  have  not  yet  received  food  handlers  training.  (The  Hennessy  program  is  a  third- 
party  audit  of  all  Air  Force  dining  facilities.  In  this  annual  contest,  each  dining  facility  is  rated 
by  a  group  of  experts  and  the  score  reflects  the  entire  scope  of  the  food  service  program.)  These 
are  a  few  examples  of  how  the  Air  Force  will  develop  a  mutually  beneficial  performance 
management  plan  by  selecting  the  metrics  rather  than  having  the  contractor  select  them.  See 
Appendix  C.l  and  Appendix  C.2  for  the  full  food  service  SDS  and  QASP. 

Therefore,  Business  Requirements  and  Advisory  Group  (BRAG)  members  should  use 
partnership  metrics  to  reflect  Air  Force  priorities  in  addition  to  relying  on  the  contractor  metrics 
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concerning  quality  control.28  The  Air  Force  also  establishes  the  targets — they  define  the 
requirements.  While  contractors  may  develop  their  own  metrics,  the  Air  Force  must  develop 
strategic  objectives  and  link  certain  metrics  to  them. 

With  Air  Force-chosen  metrics,  the  partnership  has  full-picture  reporting  by  using  information 
beyond  the  contractor’s  key  strengths.  The  metrics  should  not  report  scores  on  how  well  the 
contractor  performed  according  to  their  core  competencies.  Air  Force-chosen  metrics  may 
provide  Air  Staff  and  MAJCOMs  information  as  to  the  service  contracts’  health  and  how  well 
the  contractors  achieved  the  Air  Staff/MAJCOMs’  strategic  plans.  Contractor-chosen  metrics 
will  not  fulfill  this  purpose.  Also,  with  Air  Force-chosen  metrics,  the  staffs  will  have  a 
mechanism  to  identify  areas  requiring  resources,  policy,  or  success  stories.  Finally,  they  allow 
for  benchmarking  and  its  subsequent  best  practice  identification.  Targets  may  compare  actual 
results  to  the  following:  a)  actual  last  period;  b)  budgetary  goal;  c)  benchmark;  or  d) 
competition. 

Implementing  Air  Force-chosen  metrics  for  a  service  contract  records  its  “full  picture.”  Some 
contractors  have  used  their  own  metrics  to  claim  full  award  fees  even  though  their  metrics  may 
not  necessarily  benefit  the  Air  Force.  For  example,  if  the  Military  Family  Housing  contractor  set 
a  5-day  standard  for  Change  of  Occupancy  Maintenance,  and  then  took  60  days  for  20  units  (for 
3  average  days  per  unit),  then  the  contractor  would  claim  it  exceeded  the  standard.  However, 
without  knowing  ahead  of  time,  the  housing  maintenance  office  did  not  effectively  use  the  extra 
time  to  coordinate  their  external  efforts  (e.g.,  Traffic  Management  Flight,  new  unit  tenants,  and 
administrative  work).  The  Air  Force  would  have  benefited  if  the  contractor  accurately  forecasted 
finishing  the  Change  of  Occupancy  Maintenance,  because  then  the  housing  maintenance  office 
would  have  effectively  coordinated  related  efforts.  In  this  example,  the  contractor  would  have 
performed  well  according  to  the  contractor-chosen  metric  but  the  Air  Force  did  not  receive  much 
benefit.  See  Appendix  F.l  and  Appendix  F.2  for  the  Military  Family  Housing  maintenance  SDS 
and  QASP. 

Should  the  Air  Force  standardize  the  metrics?  No.  A  service  partnership  demands  open 
communication,  problem-solving,  and  continuous  improvement  between  the  Air  Force  Business 
Requirements  and  Advisory  Group  and  the  service  provider.  The  metrics  only  provide 
information  so  the  BRAG  may  achieve  certain  objectives.  Although  standardized  metrics  may 
identify  the  difference  between  best  practices  and  different  practices  by  reducing  ambiguity  in 
performance  information,  others  noted  the  difficulty  in  comparing  “apples-to-apples.”  For 
example,  one  may  misinterpret  a  metric  because  definitions,  standards,  and  performance  levels 
differ.29 

Metrics  commonly  change  throughout  the  contract  lifecycle.30  Even  with  an  appropriate  initial 
metric  set,  new  technology,  enhanced  strengths,  emerging  trends,  or  identified  weaknesses  may 
require  changing  it.  Metrics  also  change  as  providers  learn  more  about  their  customers.  As 
strategic  directions  and  priorities  change,  so  will  the  metrics.  Caution:  If  metrics  need  changing, 
ensure  the  revised  or  new  metrics  measure  success  in  satisfying  the  customer’s  goals. 
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Performance  management  consultants  recommend  against  using  metrics  as  an  accountability  tool 
and  for  continuous  improvement.31  Yet  commercial  firms  use  metrics  for  informative  and/or  for 
incentive  purposes  (some  firms  recommend  linking  the  metrics  with  award  fees  because 
otherwise  the  contractor  will  not  take  the  metrics  seriously).32,  33  On  the  other  hand,  some 
customers  use  metrics  to  identify  performance  strengths  and  weaknesses  and  where  to  emphasize 
cost-reducing  initiatives.  They  hesitate  to  link  metrics  to  the  award  fees  because  they  may  over¬ 
emphasize  some  areas  and  cause  unintended  consequences.  Another  challenge  may  include 
collecting  valid  and  reliable  data  (e.g.,  a  major  problem  with  Total  Quality  Management  was  the 
“pencil  whipping”  to  achieve  “good”  scores34).  One  must  practice  caution  if  the  contractor 
overachieves  in  one  area  while  underachieving  other  areas.35 

Interviews  also  addressed  barriers  to  using  performance  management  systems.  Within  public 
organizations,  some  people  noted  they  lack  financial  performance  measures  (i.e.,  having  no  profit 
to  measure).  Other  reasons  included  not  having  an  efficient  reporting  mechanism  in  place,  fear 
of  knowing  the  real  information  about  their  performance,  or  lacking  faith  in  an  accurate 
measurement  system.  The  most  common  barrier  was  the  misperception  that  using  contractor 
metrics  requires  more  time  and  effort  in  addition  to  the  random  sampling  inspections.  Further 
reasons  may  include  an  unclear  vision,  short-term  focus,  and  inability  to  link  business  planning 
to  strategy.36 


PERFORMANCE  REVIEWS 


Given  the  mutual  responsibility  of  achieving  maximum  performance,  performance  reviews 
promote  partnerships  through  open  communication.  Agendas  typically  focus  on  actual 
performance,  significant  improvements  achieved,  suggestions,  budget  status,  savings  generated, 
and  meaningful  action  items.  With  a  performance  management  plan,  the  service  provider  should 
demonstrate  historical  performance  compared  with  baseline  target  levels.  In  some  cases,  the 
service  provider  will  identify  where  they’ve  missed  the  target.  Then  they  will  address  concerns 
and  underlying  drivers.  The  service  provider  also  projects  future  performance  levels  and  issues. 
The  performance  review  may  also  include  dispute  resolution. 

Meeting  frequency  varies  within  the  public  and  private  sectors  for  each  of  the  five  service 
contracts  in  this  study.  Given  contractor  expertise,  risk,  management  attention,  priority,  and 
relevance  to  the  Air  Force’s  strategic  objectives,  meeting  frequency  varies  from  daily  to 
annually.  As  a  general  observation,  those  contract  relationships  lasting  more  than  five  years  use 
annual  performance  reviews  to  evaluate  an  award  fee.  For  more  specific  contract  monitoring, 
quarterly  meetings  typically  answer  “What  have  you  done  for  me  lately?”  During  monthly 
meetings,  partners  also  consider  the  performance  baseline  and  recommend  initiatives  for 
improving  performance.  Periodic  checks  demonstrating  the  customer’s  attention  match  the 
quarterly  and  monthly  meetings.  Commercial  and  public  organizations  also  use  daily  meetings 
for  task-specific  inspections.  The  best  frequency  is  monthly.  Weekly  metrics  may  require  more 
labor  than  necessary  and  quarterly  metrics  may  take  too  much  time  between  reports. 


14 


PARTNERSHIPS 


The  contracting  officer  may  require  a  partnership  agreement  with  the  service  provider.  The 
arrangement  promotes  achieving  mutually  beneficial  goals.  Their  agreement  should  contain 
specific  goals  and  their  objectives,  metrics,  meeting  frequency,  and  cooperation.  All  involved 
parties  sign  the  document.37  The  partnership  commits  to  specific  performance  goals  and  they 
draw  the  communication  lines,  responsibility,  and  dispute  resolution  techniques.  Note:  Some 
Air  Force  service  contracts  contain  a  120-day  walkout  clause  as  a  trial  feature  to  allow  the  Air 
Force  a  commercial  “termination”  feature  and  the  provider  a  retreat  avenue  if  the  contractor 
cannot  achieve  the  performance  levels. 

Some  contracting  officers,  contract  specialists,  quality  assurance  personnel,  and  contract 
managers  have  stated  they  anticipate  the  partnership  but  have  not  received  direction  to  use  it.  If 
they  were  directed  to  establish  a  partnership,  their  general  consensus  rests  on  having  mutually 
beneficial  goals  and  having  several  contractors  within  the  same  partnership  to  streamline 
objectives  and  efforts. 


A  CONTRACTING  OFFICER  AS  A  BUSINESS  ADVISOR 

In  order  to  participate  as  a  business  advisor  during  contract  administration,  the  contracting  officer 
(or  delegated  contract  specialist)  must  address  the  contractor’s  performance  management. 
Addressing  contractor  performance  requires  a  functional  knowledge  about  the  contract.  It  is 
necessary  to  understand  the  contributing  factors  for  each  metric.  Contract  metrics  provide 
information  to  identify  problem  areas  and  strengths.  For  example,  service  providers  may 
categorize  the  complaints  in  a  Pareto  diagram,  thus  identifying  those  problems  requiring  the  most 
attention.  A  Pareto  diagram  illustrates  the  which  category  (in  this  case,  which  complaint)  is  most 
frequent.  See  Appendix  B.2  for  an  example  using  a  Pareto  method  on  a  custodial  contract. 

The  metrics,  and  knowledge  about  them,  also  assist  in  identifying  best  practices.  In  order  to 
impart  practical  insight  for  achieving  better  performance,  the  business  advisor  must  first  know 
the  objective  performance  information.  By  comparing  the  information  to  practices  with  better 
performance  information,  the  advisor  could  then  better  serve  the  provider  with  insight.  In 
essence,  the  business  advisor  identifies  a  best  (or  better)  practice.  Without  performance 
information,  consider  how  one  might  label  a  practice  as  “best”  without  any  objective  criteria  to 
merit  the  label.  In  these  cases,  a  “best”  practice  differs  from  a  “different”  practice. 

To  prevent  this,  service  metrics  may  offer  benchmarking  as  an  added  capability  to  contract 
surveillance.  Benchmarking  efforts  typically  investigate  work  processes  as  they  compare  costs 
associated  with  either  customer  satisfaction  or  performance  levels.38  These  efforts  gather  best 
practices,  generate  new  ideas,  assist  in  strategic  planning,  aid  goal-setting,  provide  information 
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for  allocating  resources,  and  establish  current  performance  information.39  Some  compare 
productivity  level  to  service  level  while  also  reflecting  cost  information  in  order  to  obtain  best 
practice  information.40  The  best  practices  may  focus  on  opportunities  in  organization  structure, 
processes,  service  level  standards,  technology,  customer  relationship  management,  culture,  skills 
and  experience,  and  contract  management.41 

In  addition  to  identifying  strengths  and  weaknesses,  managers  commonly  use  performance 
management  for  accountability,  public  reporting,  and  program  advocacy.42  On  the  other  hand, 
internal  uses  may  include  strategic  planning,  process  evaluation,  operational  control,  and 
performance  appraisals.  Decision  makers  use  performance  management  for  two  main  purposes: 
change  management  and  business  management.43  For  changing  the  business,  they  track  trend 
data  and  measure  vision  accomplishment  according  to  targets.  For  sustaining  business 
operations,  they  focus  on  successes  and  gap  analysis  while  measuring  inputs,  outputs,  and 
outcomes.  Either  way,  the  Air  Force  shares  the  outcome’s  success  or  failure. 


CUSTOMER  COMPLAINTS 


For  sufficient  feedback,  surveys  must  show  reasons  for  customer  dissatisfaction.  However,  most 
customers  do  not  complain  or  express  their  satisfaction  or  even  dissatisfaction.44  Additionally, 
most  firms  use  quarterly  or  annual  customer  feedback  surveys,  thus  leaving  the  customer  much 
time  to  reconcile  the  complaint  before  responding  to  the  survey.45  At  best,  some  firms  monitor 
customer  complaints  at  least  monthly.46 

Therefore,  measuring  the  total  number  of  complaints  may  cause  misleading  analysis.  This 
information  does  not  indicate  why  respondents  are  complaining,  and  therefore  this  indicator  does 
not  effectively  support  the  customer-focus  objective.  Also,  in  considering  the  total  number  of 
complaints,  one  may  expect  an  initial  spike  (i.e.,  the  first  month)  and  subsequent  spikes  when 
one  encourages  facility  managers  to  use  the  complaint  system.  Besides,  providers  typically 
receive  a  low  response  rate.  Further,  QAs  and  facilities  managers  mentioned  having  “complaint 
burnout” — they  complained  about  problems  in  the  past  without  resolution  and  therefore  they  will 
not  complain  again.  Therefore,  using  only  a  total  number  of  complaints  metric  would  cause 
more  questions  than  answers.  Rather,  the  business  advisor  should  consider  the  underlying  cause 
and  strive  to  prevent  recurring  complaints.  Essentially,  the  customer  complaint  provides  data  to 
learn  more  about  customer  needs  and  expectations — not  an  achievement  score.47 

In  addition  to  identifying  the  provider’s  problem  areas,  customer  complaint  systems  should  begin 
concentrating  on  the  customer.  A  high  number  of  non-valid  customer  complaints  recurs  for  each 
service  contract  in  this  study.  For  example,  the  facility  manager  complains  about  cleaning 
services  not  performed  on  Tuesday  when  in  fact  the  custodial  contractor  was  required  to  clean  on 
Wednesday.  In  this  regard,  the  customer  does  not  know  (or  remember)  the  Service  Delivery 
Summary  for  his  or  her  facility.  As  non-valid  complaints  could  represent  a  problem  area,  QAs 
have  indicated  their  preference  to  quantify  the  percentage  of  non-valid  complaints  of  the  total 
complaints  received. 
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THE  TRADITIONAL  AIR  FORCE  CUSTOMER  COMPLAINT  PROCESS 


For  unacceptable  services,  a  base  employee,  MFH  resident,  or  QA  may  initiate  a  customer 
complaint.  Once  notified,  the  QA  fills  out  an  AF  Form  714,  Customer  Complaint  Record.  A 
complaint  is  valid  once  received  and  documented  on  the  form.  The  QA  informs  the  customer  the 
approximate  time  to  correct  the  defect.  (The  customer  must  notify  the  QA  for  uncorrected 
defects.)  Then  the  QA  forwards  the  complaint  to  the  contractor’s  Quality  Control  Inspector 
(QCI).  The  QCI  may  disagree  with  the  complaint  after  inspecting  the  site.  If  so,  the  QA  inspects 
the  defect  and  then  notifies  the  customer  for  non-valid  complaints.  If  the  QA  identifies  a  valid 
complaint,  then  the  QA  notifies  the  QCI  to  correct  the  defect.  Then  the  QA  documents  the 
incident  as  a  QCI’s  failure  to  recognize  a  valid  customer  complaint.  The  QCI  returns  the  AF 
Form  714,  complete  with  the  actions  taken,  to  the  evaluator  who  files  the  complaint  for  future 
reference. 

Data  integrity,  a  primary  concern,  becomes  suspect  when  the  customer  complaint  system 
includes  more  layers  in  the  complaint  handling  and  review  process.  People  may  act  as  filters 
which  may  either  reduce  the  number  of  complaints,  change  the  complaints  from  the  original 
cause  to  an  assumed  cause,  or  even  delay  the  total  response  time  to  the  complaint.  Automated 
reporting  (e.g.,  an  intranet)  allows  easily  accessible  data  and  reduces  the  number  of  filters 
involved  in  forwarding  information.48, 49  Therefore,  these  systems  reduce  decision  time  while 
providing  the  necessary  information.50 


A  REVOLUTIONARY  CUSTOMER  COMPLAINT  APPROACH 


As  a  means  towards  reducing  the  number  of  non-valid  complaints  and  providing  more  efficient 
communication  to  the  contractor  manager,  one  installation  automated  their  custodial  and  refuse 
collection  contracts  with  web-enabled  technology.  A  hyperlink  from  the  installation’s  homepage 
provides  facility  managers  information  such  as  their  facility’s  Service  Delivery  Summary  (SDS), 
service  frequency,  and  each  task’s  service  level.  The  site  also  contains  a  form  for  entering 
customer  complaints.  This  form  facilitates  the  customer  complaint  because  the  customer  actually 
chooses  from  a  pre-determined  complaint  list  for  the  appropriate  cause  of  customer 
dissatisfaction  (the  customer  may  also  enter  comments).  Once  the  customer  submits  a  complaint, 
the  QCI  automatically  receives  the  information  into  a  database  for  quality  analysis.  This 
automation  eliminates  those  complaints  from  the  civil  engineering  customer  service  desk  and  QA 
workload.  The  service  provider  does  not  need  to  spend  extra  time  collecting  customer 
complaints  and  then  synthesizing  the  results  into  a  report — the  web  site  updates  complaint 
information  with  each  new  incoming  complaint.  Therefore,  the  QA  and  contract  specialist  still 
have  the  necessary  performance  information  provided  to  them  in  order  to  provide  insight  towards 
continuous  improvement. 
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CHANGING  THE  SURVEILLANCE  APPROACH 


Because  AFI  63-124  has  only  been  implemented  since  1  Apr  99,  some  contracts  are  still 
compliant  with  AFM  64-108,  Service  Contracts.  Therefore,  contracting  officers  have  several 
options  in  implementing  new  surveillance  methods  (e.g.,  contractor  metrics).  The  following 
table  shows  several  options  and  their  advantages  and  disadvantages: 


Option 

Advantages 

Disadvantages 

Wait  Until  New  Award; 
Incorporate  as  Contract  Data 
Requirements  List  (CDRL) 

•  The  CDRL  is  an  easily 
identifiable  list  of  required 
reports  (i.e.,  metrics). 

•  Offerors  may  provide 
comments  on  a  draft. 

•  Offerors  may  build  the 
costs  into  their  proposal. 

•  Evolving  metrics  require 
further  contract 
modifications. 

•  Contractors  may  submit 
cost  claims  for  changes  in 
metrics. 

•  Waiting  until  awarding  the 
new  contract  may  last  as 
long  as  4  years. 

Wait  Until  New  Award; 
Incorporate  as  part  of  Quality 
Assurance  Surveillance  Plan 
(QASP) 

•  Stipulates/ensures 
receiving  metrics  from  the 
start  of  contract 
performance. 

•  Offers  “changes” 
flexibility. 

•  Offerors  may  provide 
comments  on  a  draft. 

•  Offerors  may  build  the 
costs  into  their  proposal. 

•  Contractors  do  not  submit 
cost  claims  for  changes  in 

•  Waiting  until  awarding  the 
new  contract  may  last  as 
long  as  4  years. 
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contract  surveillance. 


Modify  Existing  Contract 

•  Faster  implementation 
than  waiting  for  the  new 
award. 

•  Stipulates/ensures 
receiving  metrics. 

•  The  contractor  may  submit 
a  claim  for  additional 
work. 

•  Evolving  metrics  require 
further  modifications. 

Cost  may  exceed  benefit  if  the 
contract  is  in  its  3rd  or  4"'  year. 

Incorporate  Into  Partnership 
Agreement 

•  Fosters  partnering. 

•  Offers  flexibility. 

•  Contractor  “agrees”  to  the 
surveillance  technique. 

•  Faster  implementation 
than  waiting  for  the  new 
award. 

•  One  may  not  exist  for  that 
contract;  however,  perhaps 
it’s  time  to  create  one. 

•  May  lack  Contracting 
Officer’s  unilateral 
decision. 

•  May  be  a  weak  a 
surveillance  method. 

Introduce  As  Performance 
Review  Agenda 

•  Quick  implementation. 

•  Fosters  partnering. 

•  Offers  flexibility. 

•  Contractor  “agrees”  to  the 
surveillance  technique. 

•  Does  not  ensure  you’ll 
receive  the  information. 

•  May  lack  Contracting 
Officer’s  unilateral 
decision. 

•  May  be  a  weak 
surveillance  method. 

PERFORMANCE  PENALTIES 

Quality  Assurance  personnel  raised  a  common  concern:  With  contractor  metrics,  what  happens 
to  performance  penalties  and  Contract  Discrepancy  Reports  (CDRs)?  They  still  have  the  avenue 
of  having  the  contractor  re-perform  the  service  (otherwise  not  paying  for  that  particular  service). 
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Also,  they  still  have  the  option  of  documenting  CDRs  for  justifying  whether  or  not  to  exercise  an 
option  and  past  performance  information  for  future  source  selections. 

One  contractor  manager  perceived  the  contractor  metrics  as  a  means  towards  reporting  defects, 
and  thus  automatic  CDRs.  The  QA  addressed  the  issue  by  underscoring  the  importance  of  their 
partnership  rather  than  focusing  on  whom  to  blame  for  their  faults.  The  contractor  was  then 
willing  to  provide  metrics  at  no  cost.  Besides,  the  contractor  mentioned,  a  performance 
deduction  threat  is  not  necessarily  a  motivator  towards  achieving  maximum  performance. 
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CHAPTER  4 


CONCLUSIONS  AND 
RECOMMENDATIONS 


CONCLUSIONS 


The  metrics  are  tools  for  identifying  problem  areas,  benchmarking,  and  further  researching  best 
practices.  Metrics  are  well  suited  for  recognizing  and  advocating  the  contractor  manager’s  power 
to  motivate  line  employees  towards  continuous  improvement. 

The  metrics  don’t  achieve  maximum  performance — the  contractor  line  employees,  contractor 
managers,  QAs,  contracting  officers,  contract  specialists,  QAPCs,  functional  commanders,  and 
contracting  squadron  commanders  achieve  it.  Together,  they  make  the  performance  management 
team.  They  should  determine  the  metrics  through  buy-in  and  tailor  the  metrics  according  to  the 
Air  Staffs  or  MAJCOM’s  strategy.  They  should  also  allow  the  metrics  to  evolve  as  necessary. 
The  following  questions  are  useful  in  determining  which  metrics  to  select: 

•  What  priority  am  I  serving  in  using  my  time  for  this  metric? 

•  Why  should  I  take  the  time  to  collect  and  report  the  metrics? 

•  How  does  this  help  me  in  my  daily  operations  as  QA,  dining  facility  manager,  solid 
waste  manager,  etc.? 

•  How  will  I  have  the  authority  to  act  or  make  decisions  according  to  these  metrics? 

In  terms  of  measuring  the  service  contract’s  health  and  focusing  attention  towards  continuous 
improvement,  the  performance  management  team  should  treat  the  recommended  metrics  as  a 
baseline  to  facilitate  the  change.  Within  this  team.  Functional  Directors  and  their  QAs  may  offer 
better  indicators,  have  different  objectives,  or  have  different  performance  areas.  They  may  also 
require  receiving  more  metrics  demonstrating  their  contractors’  own  quality  control  systems. 
Metrics  will  evolve  as  personnel  actively  use,  further  develop,  and  refine  them  to  solve  problems. 
The  functional  customer  may  or  may  not  desire  to  automate  the  customer  complaint  system. 

Contractor  metrics  are  best  handled  if  they  are  included  within  the  Quality  Assurance 
Surveillance  Plan  in  addition  to  other  surveillance  methods.  A  baseline  should  be  established 
during  the  first  several  months  and  a  performance  target  may  be  identified.  This  considers  the 
transitory  nature  of  having  never  collected  the  information  to  building  it  into  the  contract  as  a 
performance  requirement.  Using  contractor  metrics  requires  other  complementing  surveillance 
methods  to  validate  performance.  Although  using  the  contractor  metrics  will  save  time 
compared  to  the  random  sampling  approach,  the  metrics  cannot  capture  every  contract 
requirement. 
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Reviewing  the  recommended  metrics  at  least  monthly  if  not  otherwise  noted  is  best  suited  for  the 
performance  management  team.  Weekly  (or  even  biweekly)  metrics  may  require  too  much  labor 
and  attention.  Quarterly  metrics  may  lose  their  capability  of  affecting  change  because  of  the  time 
delay. 


RECOMMENDATIONS 

1)  Provide  the  Service  Delivery  Summaries  (SDSs)  and  Quality  Assurance  Surveillance  Plans 
(QASPs)  to  the  Quality  Assurance  Program  Coordinators  (QAPCs)  at  all  operational 
contracting  squadrons.  (OPR:  AFLMA/LGC)  The  QAPCs  should  consider  using  the 
metrics  as  a  tool  for  identifying  problem  areas,  benchmarking,  and  further  researching  best 
practices.  See  Appendix  A.l  through  Appendix  G.2  for  the  metrics. 

2)  In  terms  of  measuring  the  service  contract’s  health  and  focusing  attention  towards  continuous 
improvement,  treat  the  recommended  metrics  as  a  baseline  to  facilitate  the  change. 

3)  Allow  installations  to  use  the  metrics  and  modify  accordingly  to  suit  their  installation’s 
mission  and  objectives.  Do  not  mandate  using  these  as  contractor  metrics. 


DISTRIBUTION:  Refer  to  attached  Standard  Form  298. 


APPENDIX  A.l 


SERVICE  DELIVERY  SUMMARY  (SDS), 
ALL  SERVICE  CONTRACTS 


Performance  Objective 

SOW  para 

Performance 

Standard  * 

Maintain  Sustained 
Workforce 

Human  Resources  Vitality 

Employee  Chum  Rate 

Generate  Innovative 
solutions 

Suggestion  Program  (Partnership) 

Number  Of 

Suggestions  Made 

Generate  And  Implement 
Innovative  Solutions 

Suggestion  Program  (Partnership) 

%  Of  Suggestions 
Implemented 

Maintain  Healthy 

Workforce 

Human  Resources  Vitality 

Number  Of  Equal 

Employment 

Opportunity 

Violations 

Maintain  Competent 
Workforce 

Human  Resources  Vitality 

Training  Hours  (Total 
Number  Of  Hours  / 
Total  Number  Of 
Employees) 

OSIIA/AFOSH 

Compliance 

Safety 

Number  Of  OSHA  Or 
AFOSH  Citations  {AFI 
91-301;  AFI 91-302; 
AFPD  91-3) 

OSHA/AFOSH 

Compliance 

Safety 

Number  Of 
Occupational  Illnesses 
And  Injuries  Resulting 
In  Lost  Days  {AFI  91- 
301,  AFI  91-302; 

AFPD  91-3) 

Subcontracting  Assistance 
For  Socioeconomic  Goals 

Socioeconomic 

%  Of  Subcontract 
Dollars  Done  With 
Small  Businesses, 
Minority-Owned,  And 
Women-Owned 
Businesses 

Compliance  With  Base 
Traffic  Regulations 

Safety 

Running  Total  Of 
Traffic  Tickets  (e.g., 
Speeding,  Accidents, 
Parking,  Seatbelts) 
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Fire  Prevention 

Safety 

Number  Of  Workers 
Who  Have  NOT 
Received  At  Least  2 
Hours  Of  Fire 

Training  Per  Year 
(AFI 32-2001) 

Installation  Security 
Compliance 

Security 

Number  Of  Workers 
Who  Do  NOT  Possess 
Proper  Identification 

*  Performance  Standard  (threshold)  will  vary  according  to  mission  needs  and  the  functional’s 
direction. 
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APPENDIX  A.2 


QUALITY  ASSURANCE  SURVEILLANCE 
PLAN  (QASP),  ALL  SERVICE  CONTRACTS 


Description  of  Services 

1.  Performance  Requirement:  Maintain  Sustained  Workforce  (Human  Resources  Vitality) 
Performance  Standard:  Employee  Chum  Rate 

Method  of  Surveillance:  Add  the  number  of  employees  hired  for  the  job  during  the  month. 
Provide  the  total  number  of  people  who  left  (quit,  retire,  separated,  removed  from  being 
assigned,  etc.).  Divide  the  total  number  of  people  who  quit  by  the  total  number  of  employees  on 
station  that  month.  This  is  the  overall  chum  rate.  This  measurement  is  a  snapshot  in  time. 


2.  Performance  Requirement:  Generate  Innovative  solutions  [Suggestion  Program 

(Partnership)] 

Performance  Standard:  Number  Of  Suggestions  Made 

Method  of  Surveillance:  Suggestions  could  focus  on  initiatives  in  reducing  costs,  saving  time, 
achieving  efficiencies  and  greater  effectiveness,  and  providing  higher  customer  service  levels. 
Suggestions  may  reflect  improvements  in  technology,  quality,  customer  responsiveness,  delivery, 
cost,  and  environmental  issues.  Suggestions  are  not  too  minor  or  trivial  to  consider.  They  are 
communicated  within  the  partnership,  not  contained  within  the  contractor’s  side.  Suggestions 
may  be  communicated  by  letter,  phone,  fax,  e-mail,  and  face-to-face.  Consider  this  metric  with 
the  “Implementing  Suggestions”  metric. 


3.  Performance  Requirement:  Generate  And  Implement  Innovative  Solutions  [Suggestion 
Program  (Partnership)] 

Performance  Standard:  %  Of  Suggestions  Implemented 

Method  of  Surveillance:  Either  the  contractor  or  a  Air  Force  representative  may  implement  a 
suggestion.  Suggestions  could  focus  on  initiatives  in  reducing  costs,  saving  time,  achieving 
efficiencies  and  greater  effectiveness,  and  providing  more  customer  service.  Suggestions  are  not 
too  minor  or  trivial  to  consider.  They  are  communicated  within  the  partnership,  not  contained 
within  the  contractor’s  side.  Suggestions  may  be  communicated  by  letter,  phone,  fax,  e-mail, 
and  face-to-face.  Suggestions  may  reflect  improvements  in  technology,  quality,  customer 
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responsiveness,  delivery,  cost,  and  environmental  issues.  Consider  this  metric  with  the 
“Generating  Suggestions”  metric. 


4.  Performance  Requirement:  Maintain  Healthy  Workforce  (Human  Resources  Vitality) 

Performance  Standard:  Number  Of  Equal  Employment  Opportunity  Violations 

Method  of  Surveillance:  Equal  Employment  Opportunity  violations  may  include  discrimination 
against  race,  sex,  religion,  creed,  and/or  ethnic  origin. 


5.  Performance  Requirement:  Maintain  Competent  Workforce  (Human  Resources  Vitality) 

Performance  Standard:  Training  Hours  (Total  Number  Of  Hours  /  Total  Number  Of  Employees) 

Method  of  Surveillance:  Add  the  total  number  of  training  hours  conducted  during  the  month. 
Divide  total  number  of  hours  by  total  number  of  employees  on  record  for  that  month. 


6.  Performance  Requirement:  OSHA/AFOSH  Compliance  (Safety) 

Performance  Standard:  Number  Of  OSHA  Or  AFOSH  Citations  (AFI  91-301 ;  A  FI  91-302; 
AFPD91-3 ) 

Method  of  Surveillance:  Provide  the  number  of  OSHA  or  AFOSH  citations  issued  and  the 
reason  for  citation. 


7.  Performance  Requirement:  OSHA/AFOSH  Compliance  (Safety) 

Performance  Standard:  Number  Of  Occupational  Illnesses  And  Injuries  Resulting  In  Lost  Days 
(AFI  91-301,  AFI  91 -302;  AFPD  91-3) 

Method  of  Surveillance:  Given  the  number  of  reported  OSHA  or  AFOSH-issued  safety  citations, 
record  the  number  of  lost  work  days  due  to  occupational  illness  or  occupational  injuries 
throughout  the  month. 


8.  Performance  Requirement:  Subcontracting  Assistance  For  Socioeconomic  Goals 

(Socioeconomic) 

Performance  Standard:  %  Of  Subcontract  Dollars  Done  With  Small  Businesses,  Minority- 
Owned,  And  Women-Owned  Businesses 
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Method  of  Surveillance:  Calculate  the  total  subcontracting  dollar  amount  for  the  month.  Break 
out  the  subcontracting  into  the  categories  listed  above.  If  you  are  unsure  of  the  subcontractor 
business  category,  then  contact  the  Contracting  Officer.  Divide  the  total  of  socioeconomic 
dollars  into  the  total  subcontracting  dollars.  Multiply  by  100. 


9.  Performance  Requirement:  Compliance  With  Base  Traffic  Regulations  (Safety) 

Performance  Standard:  Running  Total  Of  Traffic  Tickets  (e.g.,  Speeding,  Accidents,  Parking, 
Seatbelts) 

Method  of  Surveillance:  Provide  the  running  total  of  traffic  tickets  received  on  base  throughout 
the  year.  Also  provide  comments  as  to  the  type  of  violation.  Violations  may  include  speeding, 
parking,  accidents,  seatbelts,  and  weapons  on  base. 


10.  Performance  Requirement:  Fire  Prevention  (Safety) 

Performance  Standard:  Number  Of  Workers  Who  Have  NOT  Received  At  Least  2  Hours  Of  Fire 
Training  Per  Year  (AFI 32-2001) 

Method  of  Surveillance:  Add  the  total  number  of  persons  who  have  not  received  fire  training. 
The  installation  Fire  Chiefs  office  schedules  and  conducts  the  training. 


1 1 .  Performance  Requirement:  Installation  Security  Compliance  (Security) 

Performance  Standard:  Number  Of  Workers  Who  Do  NOT  Possess  Security  Markings 

Method  of  Surveillance:  Security  markings  may  include  a  personal  identification  tag  and/or  a 
vehicle  pass. 
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APPENDIX  B.l 


SERVICE  DELIVERY  SUMMARY  (SDS), 
CUSTODIAL  CONTRACT 


Performance  Objective 

SOW  para 

Performance  Standard  * 

Prevent  Recurring 
Complaints 

Customer  Responsiveness 

Pareto  Diagram  Of  Complaint 
Causes  And  Action(s)  Taken 

Resolve  Complaints 

Customer  Responsiveness 

%  Of  Contractor-Made  Follow 
Up  Calls 

Customers  Know  Valid 
From  Non-valid 

Complaints 

Customer  Responsiveness 

%  Of  Valid  Complaints  Of 

Total  Received 

Contractors  Have 
Appropriate  Training 

Training 

Number  Of  Workers  Who  Have 
NOT  Received  Appropriate 
Training 

*  Performance  Standard  (threshold)  will  vary  according  to  mission  needs  and  the  functional’s 
direction. 
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APPENDIX  B.2 


QUALITY  ASSURANCE  SURVEILLANCE 
PLAN  (QASP),  CUSTODIAL  CONTRACT 


Description  of  Services 

1 .  Performance  Requirement:  Prevent  Recurring  Complaints  (Customer  Responsiveness) 

Performance  Standard:  Pareto  Diagram  Of  Complaint  Causes  And  Action(s)  Taken 

Method  of  Surveillance:  Provide  a  monthly  classification  of  complaints  received.  Sort  the 
complaints  from  highest  number  received  to  the  lowest. 


*  Classification  may  include  the  following  for  a  Pareto  Diagram: 

General  Bathrooms 


Floors  not  vacuumed 
Floors  not  swept  or  mopped 
Floors  not  buffed 
Computer  room  floor  not  cleaned 
Carpet  not  vacuumed 
No  dusting 

Office  trash  can(s)  not  emptied 
Building  not  serviced  at  all 
Building  not  serviced  on  time 
Building  not  serviced  on  appropriate  date 
Interior  glass/mirrors  not  cleaned 
Drinking  fountains  not  cleaned 
Stairways  not  cleaned 
Walk-off  mats  not  cleaned 
Other 


No  paper  towels 
Trash  can(s)  not  emptied 
Missing  toilet  paper 
Toilets  not  cleaned 
Sinks  not  cleaned 


2.  Performance  Requirement:  Resolve  Complaints  (Customer  Responsiveness) 

Performance  Standard:  %  Of  Contractor-Made  Follow  Up  Calls 

Method  of  Surveillance:  Provide  the  number  of  complaints  received,  number  of  contractor- 
initiated  follow-up  calls,  and  the  contractor  management  action  taken.  Provide  the  percentage  of 
follow-up  calls  to  complaints  received. 
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3.  Performance  Requirement:  Customers  Know  Valid  From  Non-valid  Complaints  (Customer 
Responsiveness) 

Performance  Standard:  %  Of  Valid  Complaints  Of  Total  Received 

Method  of  Surveillance:  Take  the  total  number  of  complaints  received.  Then  identify  which 
complaints  are  non-valid.  A  non-valid  complaint  is  one  the  QA  decides  is  non-valid.  For 
example,  the  customer  complains  about  not  receiving  services  when  in  fact  the  service  is  not 
within  the  contract.  Divide  the  number  of  non-valid  complaints  by  the  total  number  of 
complaints. 


4.  Performance  Requirement:  Contractors  Have  Appropriate  Training 

Performance  Standard:  Number  Of  Workers  Who  Have  NOT  Received  Appropriate  Training 

Method  of  Surveillance:  Training  may  include  On-the-Job  Training  (OJT)  and/or  employee 
orientation. 
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APPENDIX  C.l 


SERVICE  DELIVERY  SUMMARY  (SDS), 
FULL  FOOD  SERVICE  CONTRACT 


Performance  Objective 

SOW  para 

Performance  Standard  * 

Perform  Base-Line  And 
Benchmarking  Analyses  Of  AF 
Form  1038  For  Hennessy  Program 

Hennessy  Program 
Participation — S  ingle 
Dining  Facility  Location 
(If  Applicable) 

Average  Score  Between  The 

Total  Scores  Of  Contractor- 
Rated  And  Air  Force-Rated  AF 
Forms  1038 

Perform  Base-Line  And 
Benchmarking  Analyses  Of  A  F 
Form  1038  For  Hennessy  Program 

Hennessy  Program 
Participation — Multiple 
Dining  Facility  Location 
(If  Applicable) 

Average  Score  Between  The 

Total  Scores  Of  Contractor- 
Rated  And  Air  Force-Rated  AF 
Forms  1038 

Perform  Base-Line  And 
Benchmarking  Analyses  Of  Earned 
Income 

Generating  Revenue 

Earned  Income 

Encourage  Consumption  of  Flight 
Meals 

Flight  Meals 

Earned  Income  on  Flight  Meals 

Maintain  Cumulative  Gain  (Loss) 
Under  Tolerance 

Net  Issues  to  Kitchen 

Cumulative  Gain  (Loss) 

Serve  Meals  IAW  Worldwide 

Menu  System 

Worldwide  Menu 
Compliance 

Number  Of  Meals  NOT  In 
Compliance  With  Worldwide 
Menu 

Customers  Know  Valid  From  Non- 
valid  Complaints 

Customer 

Responsiveness 

%  Of  Valid  Complaints  Of 

Total  Received 

Food  Handling  Compliance 

Training  [Directed  By  Director  Of 
Base  Medical  Services  (DBMS)] 

Sanitation 

Number  Of  Personnel  Having 
NOT  Attended  Food  Handlers 
Training 

Fast  Response  Time  to  Major  and 
Minor  Equipment  Repair 

Emergency,  Urgent,  and 
Routine — Response 

Emergency:  Number  Of 
Responses  Outside  1-Hour 
Standard 

Urgent:  Number  Of  Responses 
Outside  12-Hour  Standard 

Routine:  Number  Of 

Responses  Outside  24-Hour 
Standard 
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Fast  Completion  Time  to  Major 
and  Minor  Equipment  Repair 


Emergency,  Urgent,  and 
Routine — Completion 


Emergency:  Number  Of 
Completions  Outside  Of  24- 
Hour  Standard 

Urgent:  Number  Of 
Completions  Outside  Of  48- 
Hour  Standard 

Routine:  Number  Of 
Completions  Outside  Of  96- 
Hour  Standard 


*  Performance  Standard  (threshold)  will  vary  according  to  mission  needs  and  the  functional’s 
direction. 
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APPENDIX  C.2 


QUALITY  ASSURANCE  SURVEILLANCE 
PLAN  (QASP),  FULL  FOOD  SERVICE 

CONTRACT 


Description  of  Services 

1.  Performance  Requirement:  Perform  Base-Line  And  Benchmarking  Analyses  Of  AF  Form 
1038  For  Hennessy  Program 

Performance  Standard:  Average  Score  Between  The  Total  Scores  Of  Contractor-Rated  And  Air 
Force-Rated  AF  Forms  1038 

Method  of  Surveillance:  Once  each  month,  agree  on  a  randomly-selected  date  and  meal  period 
for  conducting  a  Hennessy  evaluation.  The  contractor  rates  the  dining  facility  (or  multiple 
facilities)  by  using  the  AF  Form  1038  and  scoring  guidelines.  Also,  the  QA  rates  the  dining 
facility  (or  multiple  facilities)  by  using  the  AF  Form  1038  and  scoring  guidelines.  See  the  next 
worksheet  (entitled  “Hennessy  Guidelines”)  for  the  scoring  system.  Match  up  objectives  within 
each  rating  sheet.  Identify  those  areas  having  the  largest  variances.  Identify  those  areas  having 
mutual  agreement  of  relatively  low  score.  Identify  those  areas  having  mutual  agreement  of 
relatively  high  score.  Discuss  the  variances,  mutual-low,  and  mutual-high  objectives  during 
performance  review. 


2.  Performance  Requirement:  Perform  Base-Line  And  Benchmarking  Analyses  Of  Earned 
Income  (Generating  Revenue) 

Performance  Standard:  Earned  Income 

Method  of  Surveillance:  Use  AF  Form  249,  Food  Service  Operations  Report  to  identify  the 
Total  Earned  Income.  This  may  include  SIK,  Cash,  MREs,  and  Satellites. 


3.  Performance  Requirement:  Encourage  Consumption  of  Flight  Meals 
Performance  Standard:  Earned  Income  on  Flight  Meals 

Method  of  Surveillance:  Use  AF  Form  467,  Monthly  Summary  of  Flight/Special  Meals  to  obtain 
the  monthly  Total  Earned  Income.  Divide  the  Total  Earned  Income  by  the  month’s  open 
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business  days.  Divide  the  Total  Earned  Income  by  the  month’s  open  business  days  for  an 
average  earned  income  per  day  (flight  meals). 


4.  Performance  Requirement:  Maintain  Cumulative  Gain  (Loss)  Under  Tolerance  (Net  Issues 
to  Kitchen) 

Performance  Standard:  Cumulative  Gain  (Loss) 

Method  of  Surveillance:  The  purpose  of  this  metric  is  to  keep  inventory  balances  at  a  minimum. 
Use  AF  Form  1119,  Monthly  Monetary  Record  to  identify  the  Adjusted  Gain  (Loss).  The 
Adjusted  Gain  (Loss)  is  an  end-of-month  inventory  balance.  The  Adjusted  Gain  (Loss)  is 
combined  with  the  Carry  Over  Last  Month  to  yield  the  Cumulative  Gain  (Loss).  The  2% 
Tolerance  is  applied  on  Earned  Income  to  yield  an  Authorized  2%  Tolerance.  The  objective  is  to 
maintain  the  Cumulative  Gain  (Loss)  within  the  Authorized  2%  Tolerance. 


5.  Performance  Requirement:  Serve  Meals  IAW  Worldwide  Menu  System 

Performance  Standard:  Number  Of  Meals  NOT  In  Compliance  With  Worldwide  Menu 

Method  of  Surveillance:  AF  Form  662  is  used  to  denote  worldwide  menu  compliance.  FSO 
approves  substitutes  as  compliance  with  the  menu.  HQ  AFSVA  approves  local  recipes. 


6.  Performance  Requirement:  Customers  Know  Valid  From  Non-valid  Complaints  (Customer 
Responsiveness) 

Performance  Standard:  %  Of  Valid  Complaints  Of  Total  Received 

Method  of  Surveillance:  Take  the  total  number  of  complaints  received.  Then  identify  which 
complaints  are  non-valid.  A  complaint  is  defined  as  a  written  comment  on  the  customer 
comment  card.  (Face-to-face  complaints  are  usually  resolved  at  the  time  of  management- 
customer  interaction.)  Divide  the  number  of  non-valid  complaints  by  the  total  number  of 
complaints.  If  reasonable,  track  the  different  kinds  of  complaints  to  see  if  trends  have  developed. 


7.  Performance  Requirement:  Food  Handling  Compliance  Training  [Directed  By  Director  Of 
Base  Medical  Services  (DBMS)]  (Sanitation) 

Performance  Standard:  Number  Of  Personnel  Having  NOT  Attended  Food  Handlers  Training 

Method  of  Surveillance:  Once  management  personnel  have  been  appropriately  trained,  they  may 
train  their  personnel. 
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8.  Performance  Requirement:  Fast  Response  Time  to  Major  and  Minor  Equipment  Repair 


Performance  Standard:  Emergency:  Number  Of  Responses  Outside  1 -Hour  Standard 

Urgent:  Number  Of  Responses  Outside  12-Hour  Standard 
Routine:  Number  Of  Responses  Outside  24-Hour  Standard 

Method  of  Surveillance:  Record  the  item  and  actual  response  time  and  completion  time  and  then 
compare  to  the  time  standards.  Response  time  is  defined  as  the  time  from  contractor  manager 
notification  to  initial  action  (phone  call,  site  visit,  etc.). 


9.  Performance  Requirement:  Fast  Completion  Time  to  Major  and  Minor  Equipment  Repair 

Performance  Standard:  Emergency:  Number  Of  Completions  Outside  Of  24-Hour  Standard 

Urgent:  Number  Of  Completions  Outside  Of  48-Hour  Standard 
Routine:  Number  Of  Completions  Outside  Of  96-Hour  Standard 

Method  of  Surveillance:  Record  the  item  and  actual  response  time  and  completion  time  and  then 
compare  to  the  time  standards.  Completion  time  is  defined  as  the  time  from  contractor  manager 
notification  to  when  the  work  is  complete. 


38 


This  Page  Intentionally 
Left  Blank 


39 


APPENDIX  D.l 


SERVICE  DELIVERY  SUMMARY  (SDS),  MESS 
ATTENDANT  CONTRACT 


Performance  Objective 

SOW  para 

Performance  Standard  * 

Perform  Base-Line  And 
Benchmarking  Analyses  Of  AF 
Form  1038  For  Hennessy  Program 

Hennessy  Program 
Participation — Single 
Dining  Facility  Location 
(If  Applicable) 

Average  Score  Between  The 

Total  Scores  Of  Contractor- 
Rated  And  Air  Force-Rated  AF 
Forms  1038 

Perform  Base-Line  And 
Benchmarking  Analyses  Of  AF 
Form  1038  For  Hennessy  Program 

Hennessy  Program 
Participation — Multiple 
Dining  Facility  Location 
(If  Applicable) 

Average  Score  Between  The 

Total  Scores  Of  Contractor- 
Rated  And  Air  Force-Rated  AF 
Forms  1038 

Perform  Base-Line  And 
Benchmarking  Analyses  Of  Earned 
Income 

Generating  Revenue 

Earned  Income 

Encourage  Consumption  of  Flight 
Meals 

Flight  Meals 

Earned  Income  on  Flight  Meals 

Customers  Know  Valid  From  Non- 
valid  Complaints 

Customer 

Responsiveness 

%  Of  Valid  Complaints  Of 

Total  Received 

Food  Handling  Compliance 

Training  [Directed  By  Director  Of 
Base  Medical  Services  (DBMS)] 

Sanitation 

Number  Of  Personnel  Having 
NOT  Attended  Food  Handlers 
Training 

*  Performance  Standard  (threshold)  will  vary  according  to  mission  needs  and  the  functional’s 
direction. 
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APPENDIX  D.2 


QUALITY  ASSURANCE  SURVEILLANCE 
PLAN  (QASP),  MESS  ATTENDANT 
CONTRACT 


Description  of  Services 

1 .  Performance  Requirement:  Perform  Base-Line  And  Benchmarking  Analyses  Of  AF  Form 
1038  For  Hennessy  Program 

Performance  Standard:  Average  Score  Between  The  Total  Scores  Of  Contractor-Rated  And  Air 
Force-Rated  AF  Forms  1038 

Method  of  Surveillance:  Once  each  month,  agree  on  a  randomly-selected  date  and  meal  period 
for  conducting  a  Hennessy  evaluation.  The  contractor  rates  the  dining  facility  (or  multiple 
facilities)  by  using  the  AF  Form  1038  and  scoring  guidelines.  Also,  the  QA  rates  the  dining 
facility  (or  multiple  facilities)  by  using  the  AF  Form  1038  and  scoring  guidelines.  See  the  next 
worksheet  (entitled  “Hennessy  Guidelines”)  for  the  scoring  system.  Match  up  objectives  within 
each  rating  sheet.  Identify  those  areas  having  the  largest  variances.  Identify  those  areas  having 
mutual  agreement  of  relatively  low  score.  Identify  those  areas  having  mutual  agreement  of 
relatively  high  score.  Discuss  the  variances,  mutual-low,  and  mutual-high  objectives  during 
performance  review. 


2.  Performance  Requirement:  Perform  Base-Line  And  Benchmarking  Analyses  Of  Earned 
Income  (Generating  Revenue) 

Performance  Standard:  Earned  Income 

Method  of  Surveillance:  Use  AF  Form  249,  Food  Service  Operations  Report  to  identify  the  Total 
Earned  Income.  This  may  include  SIK,  Cash,  MREs,  and  Satellites.  Divide  the  Total  Earned 
Income  by  the  month’s  open  business  days  for  an  average  earned  income  per  day. 


3.  Performance  Requirement:  Encourage  Consumption  of  Flight  Meals 
Performance  Standard:  Earned  Income  on  Flight  Meals 

Method  of  Surveillance:  Use  AF  Form  467,  Monthly  Summary  of  Flight/Special  Meals  to  obtain 
the  monthly  Total  Earned  Income.  Divide  the  Total  Earned  Income  by  the  month’s  open 
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business  days.  Divide  the  Total  Earned  Income  by  the  month’s  open  business  days  for  an 
average  earned  income  per  day  (flight  meals). 


4.  Performance  Requirement:  Customers  Know  Valid  From  Non-valid  Complaints  (Customer 
Responsiveness) 

Performance  Standard:  %  Of  Valid  Complaints  Of  Total  Received 

Method  of  Surveillance:  Take  the  total  number  of  complaints  received.  Then  identify  which 
complaints  are  non-valid.  A  complaint  is  defined  as  a  written  comment  on  the  customer 
comment  card.  (Face-to-face  complaints  are  usually  resolved  at  the  time  of  management- 
customer  interaction.)  Divide  the  number  of  non-valid  complaints  by  the  total  number  of 
complaints.  A  non-valid  complaints  is  one  the  QA  decides  in  non-valid  (for  example,  the 
complaint  is  about  something  outside  the  contract.)  If  reasonable,  track  the  different  kinds  of 
valid  complaints  to  see  if  trends  have  developed. 


5.  Performance  Requirement:  Food  Handling  Compliance  Training  [Directed  By  Director  Of 
Base  Medical  Services  (DBMS)]  (Sanitation) 

Performance  Standard:  Number  Of  Personnel  Having  NOT  Attended  Food  Handlers  Training 

Method  of  Surveillance:  Once  management  personnel  have  been  appropriately  trained,  they  may 
train  their  personnel. 
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APPENDIX  E.l 


SERVICE  DELIVERY  SUMMARY  (SDS), 
GROUNDS  MAINTENANCE  CONTRACT 


Performance  Objective 

SOW  para 

Performance  Standard  * 

Prevent  Recurring  Complaints 

Customer  Responsiveness 

Pareto  Diagram  Of 
Complaint  Causes  And 
Action(s)  Taken* 

Resolve  Complaints 

Customer  Responsiveness 

%  Of  Complaints 

Resolved  After  Contractor 
Correction 

Customers  Know  Valid  From 
Non-valid  Complaints 

Customer  Responsiveness 

%  Of  Valid  Complaints  Of 
Total  Received 

Provide  High-Quality  Special 
Visit  Preparation 

Distinguished  Visitors 

Recognition  From  Special 
Visit  Preparations 

*  Performance  Standard  (threshold)  will  vary  according  to  mission  needs  and  the  functional’s 
direction. 
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APPENDIX  E.2 


QUALITY  ASSURANCE  SURVEILLANCE 
PLAN  (QASP),  GROUNDS  MAINTENANCE 

CONTRACT 


Description  of  Services 


1 .  Performance  Requirement:  Prevent  Recurring  Complaints  (Customer  Responsiveness) 

Performance  Standard:  Pareto  Diagram  Of  Complaint  Causes  And  Action(s)  Taken* 

Method  of  Surveillance:  Provide  a  monthly  classification  of  complaints  received. 

Sort  the  complaints  from  highest  number  received  to  the  lowest. 

*  Classification  may  include  the  following  for  a  Pareto  Diagram: 


BASH  compliance  (not  short  enough 
around  the  flightline) 

BASH  compliance  (not  long  enough 
around  the  flightline) 

Did  not  perform  edging 

Trimming  from  edging  was  left  on  the 

grounds 

Weeds 

Did  not  mow  the  grass 
Sprinklers  went  off  at  the  wrong  time 


Lawn  not  uniformly  green 

Sprinklers  went  off  in  the  wrong  directions 

Snow  removal  (if  applicable) 

Too  much  puddling  (due  to  thatch  buildup) 

Pruning  trees 

Using  too  much  water 

Pruning  shrubs 

Drainage  (not  enough  culverts) 

Litter  patrol 
Other 


2.  Performance  Requirement:  Resolve  Complaints  (Customer  Responsiveness) 

Performance  Standard:  %  Of  Complaints  Resolved  After  Contractor  Correction 

Method  of  Surveillance:  Provide  the  number  of  complaints  received,  number  of  contractor- 
initiated  follow-up  calls,  and  the  contractor  management  action  taken.  Provide  the  percentage  of 
follow-up  calls  to  complaints  received. 


3.  Performance  Requirement:  Customers  Know  Valid  From  Non-valid  Complaints  (Customer 
Responsiveness) 
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Performance  Standard:  %  Of  Valid  Complaints  Of  Total  Received 


Method  of  Surveillance:  Take  the  total  number  of  complaints  received.  Then  identify  which 
complaints  are  non-valid.  A  non-valid  complaint  is  one  the  QA  decides  is  non- valid.  For 
example,  the  customer  complains  about  not  receiving  services  when  in  fact  the  service  is  not 
within  the  contract.  Divide  the  number  of  non-valid  complaints  by  the  total  number  of 
complaints.  A  non-valid  complaints  is  one  the  QA  decides  in  non-valid  (for  example,  the 
complaint  is  about  something  outside  the  contract.) 


4.  Performance  Requirement:  Provide  High-Quality  Special  Visit  Preparation  (Distinguished 
Visitors) 

Performance  Standard:  Recognition  From  Special  Visit  Preparations 

Method  of  Surveillance:  Record  the  number  of  compliments  received,  who  gave  it,  and  why  was 
it  given.  If  an  Air  Force  representative  (for  example,  inspector  or  contract  specialist)  receives  the 
compliment,  he  or  she  should  then  forward  it  to  the  contractor. 
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APPENDIX  F.l 


SERVICE  DELIVERY  SUMMARY  (SDS), 
MILITARY  FAMILY  HOUSING  (MFH) 
MAINTENANCE  CONTRACT 


Performance  Objective 

SOW  para 

Performance  Standard  * 

Vacancy  Is  Limited  To  3  -  5 
Days;  Contractor  Has  Excellent 
Forecasting  Ability 

COM  Program 

%  Comparison  Between 
Aggregate  #  Days  Forecasted 
Vs  Aggregate  #  Days  Actual 

Minimize  Average  Actual 
Maintenance  Downtime 
(AAMD) 

COM  Program 

Monthly  Sum  Of  Downtime 

For  All  Units  In  Contractor’s 
Custody  Divided  By  #  Of 

Units  In  Lot 

Minimize  Unacceptable 
Percentage  (UP)  Of  COM 

COM  Program 

Percentage  Of  Unready  MFH 
Units  At  Time  Of  Inspection 
Per  Month 

Fast  Response  to  Service  Calls 

Emergency,  Urgent,  and 

Routine — Response 

Emergency:  %  Of  Responses 
Outside  1-Hour  Standard 

Urgent:  %  Of  Responses 
Outside  24-Hour  Standard 

Routine:  %  Of  Responses 
Outside  5-Day  Standard 

Fast  Completion  to  Service 

Calls 

Emergency,  Urgent,  and 

Routine — Completion 

Emergency:  %  Of 

Completion  Outside  24-Hour 
Standard 

Urgent:  %  Of  Completion 
Outside  5-Day  Standard 

Routine:  %  Of  Completion 
Outside  30-Day  Standard 

Current  And  Valid  Freon 
Recovery  Systems  Certification 

Certifications 

Number  Of  Workers  NOT 
Having  Appropriate  Freon 
Certification 

Current  And  Valid  Lead-Based 
Paint  And  Asbestos 

Training 

Number  Of  Workers  NOT 
Certified  In  Lead-Based  Paint 
And  Asbestos  Handling 
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*  Performance  Standard  (threshold)  will  vary  according  to  mission  needs  and  the  functional’s 
direction. 
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APPENDIX  F.2 


QUALITY  ASSURANCE  SURVEILLANCE 
PLAN  (QASP),  MILITARY  FAMILY  HOUSING 
(MFH)  MAINTENANCE  CONTRACT 


Description  of  Services 

1.  Performance  Requirement:  Vacancy  Is  Limited  To  3  -  5  Days;  Contractor  Has  Excellent 
Forecasting  Ability  (COM  Program) 

Performance  Standard:  %  Comparison  Between  Aggregate  #  Days  Forecasted  Vs  Aggregate  # 
Days  Actual 

Method  of  Surveillance:  Contractor  submits  COM  estimate  for  each  unit  on  the  AF  Form  1219, 
BCE  Multi-Craft  Job  Order.  Once  each  month,  the  contractor  adds  up  the  total  number  of 
forecasted  days.  The  contractor  also  adds  up  the  total  number  of  actual  days  taken  to  complete 
those  units.  Chart  this  percentage  over  time. 


2.  Performance  Requirement:  Minimize  Average  Actual  Maintenance  Downtime  (AAMD) 
(COM  Program) 

Performance  Standard:  Monthly  Sum  Of  Downtime  For  All  Units  In  Contractor’s  Custody 
Divided  By  #  Of  Units  In  Lot 

Method  of  Surveillance:  At  the  end  of  each  month,  the  contractor  adds  up  the  total  number  of 
days  housing  units  were  in  custody.  This  number  is  divided  by  the  number  of  housing  units  held 
in  custody  for  an  average  actual  maintenance  downtime.  Compare  to  the  total  downtime  allowed 
for  COM  work  in  a  housing  unit.  Chart  these  numbers  over  time. 

3.  Performance  Requirement:  Minimize  Unacceptable  Percentage  (UP)  Of  COM  (COM 
Program) 

Performance  Standard:  Percentage  Of  Unready  MFH  Units  At  Time  Of  Inspection  Per  Month 

Method  of  Surveillance:  At  the  end  of  each  month,  calculate  the  percentage  of  unready  houses  at 
the  time  of  COM  inspection.  It  is  up  to  the  inspector  to  determine  if  a  unit  is  ready  or  unready  at 
inspection  time.  Chart  this  percentage  over  time. 
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4.  Performance  Requirement:  Fast  Response  to  Service  Calls  (Emergency,  Urgent,  and 
Routine) 

Performance  Standard:  Emergency:  %  Of  Responses  Outside  1 -Hour  Standard 

Urgent:  %  Of  Responses  Outside  24-Hour  Standard 
Routine:  %  Of  Responses  Outside  5-Day  Standard 

Method  of  Surveillance:  A  response  is  defined  as  the  time  from  when  the  contractor  is  notified 
until  actually  arriving  at  the  unit.  A  completion  is  defined  as  the  time  from  when  the  contractor 
was  notified  until  the  work  has  been  complete.  Service  calls  are  categorized  as  either  emergency, 
urgent,  or  routine  calls. 


5.  Performance  Requirement:  Fast  Completion  to  Service  Calls  (Emergency,  Urgent,  and 
Routine) 

Performance  Standard:  Emergency:  %  Of  Completion  Outside  24-Hour  Standard 

Urgent:  %  Of  Completion  Outside  5-Day  Standard 
Routine:  %  Of  Completion  Outside  30-Day  Standard 

Method  of  Surveillance:  A  response  is  defined  as  the  time  from  when  the  contractor  is  notified 
until  actually  arriving  at  the  unit.  A  completion  is  defined  as  the  time  from  when  the  contractor 
was  notified  until  the  work  has  been  complete.  Service  calls  are  categorized  as  either  emergency, 
urgent,  or  routine  calls. 


6.  Performance  Requirement:  Current  And  Valid  Lead-Based  Paint  And  Asbestos  (Training) 

Performance  Standard:  Number  Of  Workers  NOT  Certified  In  Lead-Based  Paint  And  Asbestos 
Handling 

Method  of  Surveillance:  Identify  the  number  of  workers  NOT  trained  and  certified  in  both 
asbestos  and  lead-based  paint  abatement.  Workers  may  belong  to  the  contractor  and/or 
subcontractor(s). 

7.  Performance  Requirement:  Current  And  Valid  Freon  Recovery  Systems  Certification 
(Certifications) 

Performance  Standard:  Number  Of  Workers  NOT  Having  Appropriate  Freon  Certification 

Method  of  Surveillance:  Identify  the  number  of  workers  NOT  trained  and  certified  in  Freon. 
Workers  may  belong  to  the  contractor  or  subcontractor(s). 
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APPENDIX  G.l 


SERVICE  DELIVERY  SUMMARY  (SDS), 
REFUSE  AND  RECYCLING  CONTRACT 


Performance  Objective 

SOW  para 

Performance  Standard  * 

Prevent  Recurring 
Complaints 

Customer  Responsiveness 

Pareto  Diagram  Of  Complaint 

Causes  And  Action(s)  Taken* 

Resolve  Complaints 

Customer  Responsiveness 

%  Of  Complaints  Resolved  After 
Contractor  Correction 

Customers  Know  Valid 
From  Non-valid 

Complaints 

Customer  Responsiveness 

%  Of  Valid  Complaints  Of  Total 
Received 

Achieve  Diversion  Rate  By 
Promoting  Recycling 

Solid  Waste  Reduction 

Total  Amount  Recycled  /  (Total 
Amount  Recycled  Plus  Total 

Amount  Disposed) 

*  Performance  Standard  (threshold)  will  vary  according  to  mission  needs  and  the  functional’s 
direction. 


This  Page  Intentionally 
Left  Blank 
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APPENDIX  G.2 


QUALITY  ASSURANCE  SURVEILLANCE 
PLAN  (QASP),  REFUSE  AND  RECYCLING 

CONTRACT 


Description  of  Services 

1 .  Performance  Requirement:  Prevent  Recurring  Complaints  (Customer  Responsiveness) 

Performance  Standard:  Pareto  Diagram  Of  Complaint  Causes  And  Action(s)  Taken* 

Method  of  Surveillance:  Provide  a  monthly  classification  of  complaints  received.  Sort  the 
complaints  from  highest  number  received  to  the  lowest. 

*  Classification  may  include  the  following  for  a  Pareto  Diagram: 

General  Military  Family  Housing  (MFH)  Trash  Bins 


Too  much  noise 
Personnel  Attitude 

The  pickup  time  (recycling  and/or  trash) 
was  changed  and  I  was  not  informed 


MFH  trash  bin  not  picked  up  at  all 

Didn’t  pick  up  all  of  the  trash 

MFH  trash  bin  not  picked  up  on  time 

Trash  left  around  the  MFH  trash  bin 

Odor  around  MFH  trash  bins 

Trash  bin  lid  not  closed 

Not  placing  MFH  trash  bin  upright 

Trash  bin  was  missing 

Would  a  pick  up  on  a  different  day 

Other 


Dumpsters 

Dumpster  not  picked  up  on  time 
Trash  left  around  the  dumpster 
Odor  around  dumpsters 
Dumpster  was  too  full 
Lid  not  closed 

Dumpster  was  not  picked  up  at  all 
Didn’t  pick  up  all  of  the  trash 
Would  like  trash  picked  up  on  different  day 
Other 


Military  Family  Housing  (MFH)  Recycling  Bins 

MFH  recycling  bin  not  picked  up  at  all 
Did  not  pick  up  all  of  the  recycling 
MFH  recycling  bin  not  picked  up  on  time 
Debris  left  around  the  MFH  recycling  bin 
Odor  around  MFH  recycling  bins 
Recycling  bin  lid  not  closed 
Not  placing  MFH  recycling  bin  upright 
Recycling  bin  was  missing 
Would  like  pick  up  on  a  different  day 
Other 
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2.  Performance  Requirement:  Resolve  Complaints  (Customer  Responsiveness) 

Performance  Standard:  %  Of  Complaints  Resolved  After  Contractor  Correction 

Method  of  Surveillance:  Provide  the  number  of  complaints  received,  number  of  contractor- 
initiated  follow-up  calls,  and  the  contractor  management  action  taken.  Provide  the  percentage  of 
follow-up  calls  to  complaints  received. 


3.  Performance  Requirement:  Customers  Know  Valid  From  Non-valid  Complaints  (Customer 
Responsiveness) 

Performance  Standard:  %  Of  Valid  Complaints  Of  Total  Received 

Method  of  Surveillance:  Take  the  total  number  of  complaints  received.  Then  identify  which 
complaints  are  non-valid.  A  non-valid  complaint  is  one  the  QA  decides  is  non-valid.  For 
example,  the  customer  complains  about  not  receiving  services  when  in  fact  the  service  is  not 
within  the  contract.  Divide  the  number  of  non-valid  complaints  by  the  total  number  of 
complaints. 


4.  Performance  Requirement:  Achieve  Diversion  Rate  By  Promoting  Recycling  (Solid  Waste 
Reduction) 

Performance  Standard:  Total  Amount  Recycled  /  (Total  Amount  Recycled  Plus  Total  Amount 
Disposed) 

Method  of  Surveillance:  Divide  the  Total  Amount  Recycled  by  the  Total  Amount  Recycled  Plus 
Total  Amount  Disposed.  Each  quarter,  the  contractor  submits  the  total  tonnage  of  non-hazardous 
waste  disposed.  The  contractor  also  submits  the  total  tonnage  of  recycled.  This  is  a  “modified” 
diversion  rate  because  it  focuses  directly  on  the  contractor’s  portion  of  recycling  and  refuse 
disposal.  The  rate  disregards  other  factors  included  in  the  installation’s  diversion  rate  (DRMO 
proceeds.  Services  recycling,  construction  debris,  and  etc.). 
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